Conversation-based foreign language learning method using reciprocal speech transmission through speech recognition function and tts function of terminal

ABSTRACT

A method for foreign language learning between a learner and a terminal, based on video or audio containing foreign language, particularly, to a conversation-based foreign language learning method using a speech recognition function and a TTS function of a terminal, a learner learns a foreign language in a way that: the terminal reads a current learning target sentence to the learner to allow the learner to speak the current learning target sentence after the terminal, when speech input by the learner in a speech waiting state of the terminal is the same as the current learning target sentence or belongs to the same category as the current learning target sentence; and the terminal and the learner alternately speak sentences one-by-one when the speech input by the learner is the same as the next sentence of the current learning target sentence or belongs to the same category as the next sentence.

TECHNICAL FIELD

The present invention relates to a conversation-based foreign languagelearning method that enables foreign language learning through naturalconversation between a learner and a terminal, such as a smartphone, atablet computer, a notebook computer, a talking toy, an AI speaker, andthe like, based on video or audio content (hereinafter referred to as“foreign language content” or “content”) containing foreign languagesound expressions, such as movies, dramas, pop songs, news, and thelike, via speech transmission using a speech recognition function and atext-to-speech (TTS) function of the terminal. More particularly, thepresent invention relates to a conversation-based foreign languagelearning method using a speech recognition function and a text-to-speech(TTS) function of a terminal, by which a learner learns a foreignlanguage in a way that: the terminal executes a preset command inresponse to speech input by the learner in a speech waiting state forspeech recognition and enters the speech waiting state again upondetermining that the speech input by the learner corresponds to thepreset command; the terminal enters the speech waiting state afterreading the next learning target sentence of a current learning targetsentence to the learner such that the learner speaks the next learningtarget sentence after the terminal, upon determining that the speechinput by the learner is the same as the current learning target sentenceor belongs to the same category as the current learning target sentence,and enters the speech waiting state after reading the current learningtarget sentence to the learner again such that the learner speaks thecurrent learning target sentence after the terminal, upon determiningthat the speech input by the learner does not belong to the samecategory as the current learning target sentence; and the terminal andthe learner alternately speak sentences one-by-one when the speech inputby the learner is the same as the next learning target sentence of thecurrent learning target sentence or belongs to the same category as thenext learning target sentence, whereby the learner can naturally learnthe foreign language through interactive conversation with the terminalvia speech transmission without touch, if any, a screen of the terminalor by minimizing the touch of the screen of the terminal to exceptionalcases.

BACKGROUND ART

In general, various foreign language learning methods have been used tolearn foreign languages using foreign language content and devicesadapted to read content, such as PCs, notebook computers, tabletcomputers, smartphones, and the like, and recently, as speechrecognition technology and TTS technology can be applied to suchdevices, various foreign language learning methods adopting suchtechnologies have been proposed.

In addition, companies, such as Google and Samsung, have developedprograms that combine TTS technology with speech recognition technology,such as Google Assistant and Samsung Bixby, to allow users to ask aquestion or to execute a function of a terminal by speaking to theterminal.

However, the foreign language learning method using the foreign languagecontent is mainly aimed at allowing a learner to enjoy video or audiocontent executed by a terminal and is not carried out throughconversation between the learner and the terminal via speechtransmission, thereby providing an insufficient opportunity for thelearner to speak by himself or herself. Moreover, since the programadopting the speech recognition function to make the learner execute orsearch for a certain function on the terminal with words is notbasically aimed at proper foreign language learning, it is not easy forthe learner to have a conversation using the program unless the learnerhas already studied a corresponding foreign language to some extent tocontinue a conversation in the foreign language, and even if a learnerhas an ability to communicate to some extent, the program is notsuitable for foreign language learning since the program has alimitation in conversation topics and does not guarantee accuracy ofexpressions used by the learner.

Prior documents of the present invention include KR Patent RegistrationNo. 10-0954498, No. 10-1125572, No. 10-1427528, and the like.

DISCLOSURE Technical Problem

The present invention is developed to solve such problems ofconventional techniques. The inventors conceived technical ideas that alearner can be guided to learn a foreign language by having a continuousconversation with a terminal through speech transmission using speechrecognition technology and TTS technology to have an opportunity tospeak the foreign language with his or her own voice as much as possibleand to learn correct foreign language expressions by continuinginteractive conversation with the terminal through speech transmissionbased on foreign language content corresponding to a collection ofcorrect foreign language expressions, and completed the presentinvention by sequentially solving various problems in the course ofspecifying the technical ideas.

In order to allow a learner to learn a foreign language throughconversation with a terminal, such as a smartphone, a tablet computer, anotebook computer, a talking toy, an AI speaker, and the like, which hasa speech recognition function and a text-to-speech (TTS) function, usingforeign language content through application of the speech recognitionfunction and the text-to-speech (TTS) function, problems to be solved bythe present invention are as follows:

(1) In order to allow a learner to learn a foreign language throughcommunication with a terminal instead of pressing a button on theterminal, it is necessary to allow the terminal to enter a speech inputstate in response to speech input by a learner (hereinafter referred toas “speech waiting state” or “waiting state”) such that the learnerinputs the speech through the terminal substantially in all situationsthat request the learner to press the button in conventional techniques.However, since it is necessary for the learner to press a button whichmakes the terminal return back to the waiting state in a program adaptedto execute the speech recognition function, this requirement does notmeet the objective of the present invention to allow transmission ofmost commands through learner voice. Therefore, in order to completelyachieve the objective of the present invention, the program includescreation of the waiting state for the learner to transmit commands withvoice among content to be executed by voice commands of the learner.

(2) When the learner inputs a voice in the speech waiting state of theterminal, the terminal is required to recognize meanings of each voicecommand spoken by the learner and to react through foreign languagespeech in response to the voice command or to execute a functioncorresponding thereto. As a result, foreign language learning is carriedout through interactive conversation between the learner and theterminal via speech transmission, thereby providing an opportunity forthe learner to speak the foreign language as much as possible whilepreventing a problem that the learner stops conversation with theterminal in the course of touching a screen of the terminal. To thisend, it is necessary for the terminal to store separate commandexpressions to be transmitted to the terminal by the learner throughlearner voice and to execute a function corresponding to an expressionspoken by the learner upon determining that the expression correspondsto a preset command.

(3) In order to allow the learner to learn a foreign language throughinteractive conversation with the terminal between the learner and theterminal via speech transmission using foreign language expressions inthe foreign language content, there is a need for a learning method inwhich the terminal plays a learning target sentence with a content soundor a TTS sound such that the learner speaks the learning target sentenceafter the terminal. By such a learning method, the learner is able tounderstand a correct pronunciation method and correct foreign languageexpressions. In the course of performing such foreign language learning,it is necessary to allow the terminal to react in various ways based ondetermination as to whether the learner correctly speaks a learningtarget sentence. That is, if the learner correct speaks the learningtarget sentence, it is determined that the learner has sufficientlycarried out foreign language learning and the learner is guided to learnthe next learning target sentence, and if the learner fails to correctlyspeak the learning target sentence, the learner is guided to carry outlearning of the current learning target sentence. Here, upon determiningwhether the sentence spoken by the learner matches the current learningtarget sentence, it is very difficult to achieve an accuratedetermination result due to limitations of foreign languagepronunciation ability of the learner and recognition ability of thespeech recognition program. Thus, if the learner is requested to performforeign language learning based on the current language alone, thelearner can give up learning of other foreign language sentences and canconsume an excessive time due to learning a single foreign languagesentence, thereby making it difficult to achieve foreign languagelearning through natural conversation. Therefore, it is necessary tosolve such problems in order to achieve foreign language learning in amore natural conversation format.

(4) In order to achieve natural conversation-based foreign languagelearning, there is a need for a means for foreign language learningthrough interactive conversation in which the terminal and the learneralternately speak sentences one by one. However, since foreign languagelearning cannot be sufficiently achieved only through a process in whichthe learner listens to and speaks a current sentence or severalsentences executed by the terminal, there is a need for a foreignlanguage learning method that allows the learner to have an actualinteractive conversation with the terminal using an ability obtainedthrough such a foreign language learning process and to continue foreignlanguage learning based on a result of the conversation.

(5) In a typical method in which the learner presses a button (includinga button for making the terminal enter the speech waiting state), sincethe learner waits until a button for execution of the next learningtarget sentence by the terminal is pressed (for example, until video oraudio execution for one learning target sentence is completed), based ondetermination when the button should be pressed, and presses the buttonto allow the terminal to execute the next learning target sentence,there is no need for consideration of a time at which the learnerpresses the button. However, in order to replace manipulation ofpressing the button by speech transmission of the learner, it isnecessary to allow the terminal to enter the waiting state for receivingspeech of the learner when the button is pressed by the learner in therelated art. Accordingly, foreign language learning can be carried outthrough more natural conversation if it is possible to calculate anddeliver information about the time when the terminal should be turnedinto the waiting state.

(6) There are some differences between foreign language expressionsrealized by sounds or subtitles in foreign language content (since thereare cases where audio content is also displayed together with subtitles,such as lyrics and lines, all cases displayed as subtitles regardless ofthe form of lyrics or lines will be referred to as subtitles.) and soundor text expressions generated by the learner speaking the foreignlanguage expressions after the sounds or the subtitles. A means forovercoming such differences enables foreign language learning in a morenatural conversation format. In comparison of texts of both sentences ina process of comparing the content of speech input by the learner andthe sentence content of the foreign language content, the followingpoints should be considered. Even if the subtitles in the foreignlanguage content are exactly correct, the sentence expressions in theform of text contained in the subtitles can be inconsistent withsentence expressions generated by a speech recognition function of theterminal from several viewpoints. For example, the two sentences aredifferent depending upon uppercase and lowercase characters for eachprogram, punctuation marks, such as question marks, exclamation marks,quotation marks, and parentheses, and use of various unit indicators,such as $, kg, and %, expression methods of numbers or proper nouns, andthe like, which are provided by speech recognition programs used in theterminal. In general, since a computer recognizes a blank as a singlecharacter in determining identity between two character strings, adetermination result as to the identity between the character stringsvaries depending on the presence of the blank and the location andnumber of the blank. As a result, even though the text in the subtitlesand the text generated through the speech recognition function havesubstantially the same meaning, there are cases where these texts arerecognized as being not identical. If such a difference exists, theterminal treats the learner as having made an incorrect expression eventhrough the learner makes a correct foreign language expression.Therefore, it is possible to achieve foreign language learning in a morenatural conversation format through the means for overcoming suchdifferences.

(7) In order to allow more natural and easier foreign language learningthrough interactive conversation between the learner and the terminal,there is a need for additional means for compensating for difficulty ofspeaking after the learner listens to a learning target sentenceprovided as a content sound and understands the learning targetsentence.

In the course of foreign language learning through speech transmissionbetween the learner and the terminal while minimizing contact with thebutton or the screen of the terminal, confusion can occur between voicecommands given by the learner to the terminal and expressions of thelearning target sentence, causing an unexpected response of the terminalto the learner. An additional means for preventing confusion between thevoice commands and the learning target sentences enables more naturalforeign language learning of the learner through interactiveconversation with the terminal via speech transmission.

According to pronunciation ability, the learner can speak all or part ofthe learning target sentence (including all forms of phonemes,syllables, and words) so as not to be recognized as the learning targetsentence. Thus, there is a need for a means for clarifying a part of thesentence often misspelled by the learner. Such a means allows thelearner to intensively learn the frequently misspelled part of thelearning target sentence, thereby enabling improvement in foreignlanguage pronunciation ability of the learner while enabling foreignlanguage learning through natural conversation.

(8) It is necessary to prevent occurrence of a situation in which thelearner finds and presses the button for making the terminal enter thewaiting state since the terminal does not enter the waiting state andthe learner cannot input speech even though the learner is required toinput the speech. In general, in the waiting state of the terminal inthe speech recognition program, the terminal sends a special soundsignal to the learner or displays an icon of a specific shape on thescreen together with the sound signal to notify that the terminal isready to accept the speech input by the learner, and is then maintainedin this state for a few seconds (generally, 5 to 7 seconds). Then, ifthere is no voice input, the terminal changes the shape of the icon anddoes not return to the waiting state unless the part is touched again.In order to achieve the objectives of the present invention, it isnecessary to prevent occurrence of such a situation in which the waitingstate is lost, and if the waiting state is lost, it is necessary todevise an additional means for converting the terminal into the waitingstate without finding and pressing the button.

(9) In order for the learner to continue foreign language learningthrough interactive conversation with the terminal without losinginterest, there can be a need for an additional means that can inspirelearner's motivation to learn the foreign language and allows parentsand teachers to check the learner's achievements by guiding the learnerto speak a correct foreign language expression such that the terminalperforms a predetermined function or by reflecting and displayinglearning performance of the learner in numerical values in the caseswhere the learner carries out the next stage of foreign languagelearning by speaking a correct expression in learning target content,where the learner patiently continues foreign language learning for aconsiderable period of time, and the like. When the method according tothe present invention is realized through a server, such a means caninduce competition through comparison among several learners, therebyfurther inspiring learners' motivation to achieve foreign languagelearning. Therefore, the means for calculating and displaying thelearning performance of the learners as described above may be includedin the objectives of the present invention.

(10) In the method according to the present invention, when the learnerspeaks a command or a learning target sentence with his/her own voice,there can continuously occur a situation in which the terminaldetermines whether the speech input by the learner matches a sentence ofa command or a learning target sentence previously stored in theterminal. In this process, the terminal executes the speech recognitionfunction by recording the speech input by the learner, extractingnecessary data from the recording result, and comparing the data withcomparison data in a speech recognition algorithm. Then, if the speechinput by the learner is not stored after such comparison, the recordingresult can be removed. Here, the result of recording what the learnerspeaks with his/her own voice with an intention of correctly reading acertain sentence is stored corresponding to the learning target sentenceand data is extracted from the recording result so as to be used forvarious purposes including performing the speech recognition function.Therefore, there is a need for a means that can store the recordingresult corresponding to the learning target sentence instead of removingthe recording result to be used as data for more efficient speechrecognition and foreign language learning.

In addition, there is a need for a means that allows a speechrecognition device to recognize what the learner speaks with anintention of correctly reading a learning target sentence and can storetext of the recognized result separately from the learner's voice so asto be used for various purposes mentioned above.

(11) Applications implementing the present invention (hereinafter,referred to as “apps”) employ foreign language content and can requirepermission of a copyright holder for use of the foreign language contentdepending upon the foreign language content. Since the foreign languagecontent is not originally created with the purpose of foreign languagelearning through natural conversation format, there can be cases whereit is necessary to obtain permission from the copyright holder for useof such content for foreign language learning. Although there is a lawthat makes it possible to use such content for educational purposeswithout permission of the copyright holder, it is often unclear whetherit is necessary to obtain permission of the copyright holder in the caseof commercial use of such content. Therefore, there is a need for ameans that can prevent implementation of the present invention frombecoming illegal by confirming in the course of implementing the presentinvention whether the foreign language content given as a learningtarget is implemented under the permission of the copyright holder.

(12) For foreign language learning, there is a need for subtitles whichmake a learner understand meanings of a foreign language to be learnedby the learner. Therefore, there is a need for means that provide aplurality of such subtitles in different languages, allow the pluralityof subtitles to be displayed on a screen for learning, and allow thelearner to use the subtitles in a language other than his/her nativelanguage for learning of a second foreign language, whereby learners ofdifferent languages can use the terminal together while allowing thelearners to learn a second foreign language.

Technical Solution

In accordance with one aspect of the present invention, there isprovided a conversation-based foreign language learning method using aspeech recognition function and a TTS function of a terminal throughinteractive conversation between a learner and a terminal based on videoor audio foreign language content containing a plurality of foreignlanguage learning target expressions, wherein the terminal informs thelearner of a speech waiting state of the terminal with a sound signaland/or through a screen thereof with an image indicating the speechwaiting state of the terminal, and, when the learner inputs speech tothe terminal in the speech waiting state, the terminal respondsaccording to the content of the input speech and returns to the speechwaiting state, whereby the learner can perform foreign language learningthrough interactive conversation between the learner and the terminalvia speech transmission without touch of, if any, a screen or byminimizing the touch of the screen to exceptional cases.

The terminal may previously store voice commands to be delivered fromthe learner to the terminal in a server or the cloud and may execute acorresponding command upon determining that the speech input by thelearner in the speech waiting state matches a specific command throughcomparison with the stored command.

In order to prevent unexpected reaction of the terminal due to confusionbetween the voice command given by the learner to the terminal andexpressions of a learning target sentence, an identifier may be added toall or part of the voice commands delivered from the learner to theterminal so as to prevent confusion with the learning target sentence.

In the conversation-based foreign language learning method, foreignlanguage learning may be repeatedly carried out by allowing the terminalto provide the speech waiting state after reading the next learningtarget sentence of a current learning target sentence to the learnersuch that the learner speaks the next learning target sentence after theterminal, upon determining that the speech input by the learner is thesame as a current learning target sentence or belongs to the samecategory as the current learning target sentence, or to provide thespeech waiting state after reading the current learning target sentenceto the learner again such that the learner speaks the current learningtarget sentence after the terminal, upon determining that the speechinput by the learner is not is the same as the current learning targetsentence or does not belong to the same category as the current learningtarget sentence.

In the conversation-based foreign language learning method, foreignlanguage learning may be repeatedly carried out by allowing the terminaland the learner to alternately speak sentences one-by-one upondetermining that the speech input by the learner is the same as the nextlearning target sentence of the current learning target sentence orbelongs to the same category as the next learning target sentence.

A period of time until the terminal enters the speech waiting state maybe set in consideration of a length of a corresponding foreign languagelearning target sentence and a period of time for which thecorresponding foreign language learning target sentence is executed inthe foreign language content.

In determination as to whether the speech input by the learner is thesame as learning target expressions in the foreign language content orbelongs to the same category as the learning target expressions, textsof both sentences may be compared assuming that punctuation marks, suchas a question mark (?), an exclamation point (!), a comma (,), a period(.), a quotation mark (“ ”), a hyphen (-), a center point (⋅),parentheses {( )}, curly braces ({ }), square brackets ([ ]), or spacesincorrectly added at the beginning and end of sentences, and spacesoverlapping each other are absent; and that an expression of numbers inArabic numerals and an expression of numbers with letters; an expressionof following verbs or auxiliary verbs in abbreviated forms using anapostrophe (') after a subject and an expression of original formsthereof without using abbreviations; an expression of negative sentencesin abbreviated forms using an apostrophe (') and an expression oforiginal forms thereof without using abbreviations; “going to” and“gonna”; “want to” and “wanna”; “got to” and “gotta”; or a sentence usedby distinguishing between uppercase and lowercase characters and asentence used without distinction therebetween are regarded as beingexactly or partially identical to each other.

When learning with respect to a foreign language learning targetsentence is repeated due to failure of the learner to continuously inputspeech that belongs to the same category as the foreign languagelearning target sentence or when the learner inputs simple speech, suchas “yes” and “okay”, or a preset voice command, such as “you speak thenext learning target sentence”, the terminal may stop reading of thecurrent learning target sentence and may proceed to the next learningtarget sentence to allow the learner to learn the next learning targetsentence.

A terminal having a screen may display a current learning targetsentence together with the next learning target sentence.

The conversation-based foreign language learning method further includesa process in which the terminal reads a part (one or multiple words orcharacters) of a specific sentence to the learner with a TTS sound amonglearning target sentences, enters the speech waiting state and informsthe learner of the speech waiting state to allow the learner to speakthe part of the specific sentence after the terminal; and the terminalproceeds to and read a part of the next learning target sentence to thelearner with a TTS sound enters the speech waiting state and informs thelearner of the speech waiting state to allow the learner to speak thepart of the next learning target sentence after the terminal upondetermining that speech input by the learner is the same as the part ofthe learning target sentence provided through TTS or belongs to the samecategory as the part of the learning target, and the terminal reads thesame part of the specific sentence to the learner with the TTS sound toallow the learner to speak the same part of the specific sentence afterthe terminal, upon determining that the recognition result is not thesame as the part of the learning target sentence provided through TTS ordoes not belong to the same category as the part of the learning target.

The terminal may read native language subtitles to the learner with aTTS sound corresponding to a foreign language content sound for thelearning target sentence before the terminal provides the foreignlanguage content sound or a TTS sound for the learning target sentenceto the learner.

The terminal may read the learning target sentence to the learner with aTTS sound before the terminal provides a foreign language content soundfor the learning target sentence to the learner.

The terminal may read native language subtitles to the learner with aTTS sound corresponding to a foreign language content sound for thelearning target sentence or may read the learning target sentence to thelearner with a TTS sound before the terminal provides the foreignlanguage content sound or the TTS sound for the learning target sentenceto the learner.

When the speech waiting state of the terminal disappears, the learnermay be guided to press one or more buttons on a remote controller(including a pen of Samsung Galaxy Note) or earphones without touchingthe screen on the terminal, or may be guided to use another softwareprogram for speech recognition, such as Samsung Bixby or GoogleAssistant, so as to activate the speech waiting state.

Further, in a process of allowing the learner to learn a foreignlanguage through interactive conversation using the foreign languagecontent, performance of foreign language learning of the learner may beevaluated and displayed numerically.

When the terminal proceeds to the next learning target sentence toperform foreign language learning based on determination that the speechinput by the learner is exactly the same as the current learning targetsentence, speech recording data of the learner may be storedcorresponding to the learning target sentence such that speech recordingdata generated by exactly speaking a specific sentence can be preservedand used as needed.

Upon determining that speech input by the learner is not exactly thesame as a learning target sentence, text converted from the speech ofthe learner by a speech recognition function of the terminal may bestored corresponding to the learning target sentence or a sentencenumber thereof.

Upon determining that speech input by the learner is not exactly thesame as a learning target sentence, a difference between the textconverted from the speech of the learner by the speech recognitionfunction of the terminal and the learning target sentence may beextracted and informed to the learner.

Upon determining that speech input by the learner is not exactly thesame as a learning target sentence, a difference between the textconverted from the speech of the learner by the speech recognitionfunction of the terminal and the learning target sentence may be storedand sentences in which the learner most frequently makes mistakes may beranked based on the stored data to provide all or some of the sentencesto the learner.

In addition, the method according to the present invention may include aprocess of displaying a message asking to execute foreign languagelearning after storing foreign language content with permission of acopyright holder in a storage device, followed by prohibiting foreignlanguage learning, upon determining that there is no foreign languagecontent with permission of the copyright holder in the storage device.

The method may include at least one selected from the group consistingof a process of recording and storing data obtained by exactly speakinga learning target sentence by the learner to use the data as data forimproving speed and accuracy of speech recognition in comparison with acontent sound for the corresponding sentence, a process of recording andstoring data obtained by exactly speaking a learning target sentence bythe learner to use the data as data for improving speed and accuracy ofspeech recognition in comparison with previous recording data of thelearner, a process of recording and storing data obtained by exactlyspeaking a learning target sentence by the learner to use the data asassistant data in comparison with data for speech recognition stored inthe storage device by a speech recognition system, and combinationsthereof.

The terminal may be provided with a plurality of subtitles created usinga plurality of languages including a native language of the learner suchthat subtitles created using a foreign language can be displayed on thescreen of the terminal by learner selection, followed by allowing theterminal to read all or part of a certain subtitle to the learner amongthe subtitles displayed on the screen using a TTS function of theterminal and to respond to a recognition result generated by recognizingspeech input by the learner through a speech recognition function of theterminal when the learner inputs the speech after the terminal.

The present invention provides an application program including theconversation-based foreign language learning method according to thepresent invention.

Advantageous Effects

The conversation-based foreign language learning method using a speechrecognition function and a TTS function of the terminal according to thepresent invention allows a learner to have interactive conversation witha terminal using one sentence in foreign language content contained inthe terminal while using the terminal like a native speaker friend orteacher accompanying with the learner, and allows the terminal torespond to speech of the learner such that the learner repeatedly speaksa foreign language sentence until the learner speaks a correct foreignlanguage sentence by immediately determining whether the foreignlanguage sentence spoken by the learner is correct, thereby making itpossible for the learner to learn foreign languages naturally. Themethod according to the present invention provides many advantages ascompared with the case of receiving a particularly expensive privateschool or private tutoring. As a result, since the learner is notrestricted by place and time, the learner can improve an ability tospeak a foreign language through repeated learning. The method accordingto the present invention allows the learner to play a role of aprotagonist in scenes while repeatedly watching or listening to his orher favorite movies, dramas, and pop songs or to repeat learning to anextent that they can memorize lines of famous scenes.

In order to evaluate through the speech recognition function that thelearner correctly speaks a learning target sentence, the learner isrequired to speak most identically to a foreigner's voice. In order toreceive such an evaluation, the learner is guided to make an effort tocorrect his or her pronunciation, thereby achieving improvement inforeign language learning.

According to the present invention, in a first stage of animplementation process, the learner is guided to understand the overallcontent of foreign language content while watching video or listening toaudio, and is guided to practice speaking of each learning targetsentence in the foreign language content to be similar to learningtarget sentences generated with a content sound or a TTS sound whilelistening to the learning target sentences generated with the contentsound or the TTS sound, after or as soon as the learner understands theoverall content. Such processes for foreign language learning throughinteractive conversation may be performed while the learner walks on astreet using earphones regardless of location or may be performed usinga terminal not including a screen or in a situation making it difficultfor the learner to watch, if any, the screen of the terminal at night,whereby the learner can more naturally enjoy foreign language learningthrough interactive conversation by understanding media content, such asmovies, dramas, pop songs, and news, in a short period of time andmemorizing the media content, depending on an effort level of thelearner. Since such learner's achievements can be easily confirmed bythe learner himself/herself or by learner's parent by the method ofdisplaying performance of foreign language learning according to thepresent invention, thereby further improving the effect of foreignlanguage learning.

According to the present invention, learner's voice data correspondingto a specific sentence generated in the course of performing foreignlanguage learning is stored and utilized, whereby the learner can speaka foreign language more correctly and can perform foreign languagelearning through more natural conversation.

According to the present invention, speech data generated by correctlyreading learning target sentences by the learner are recorded and storedas data for improvement in speed and accuracy of speech recognitionthrough comparison with content sounds with respect to the correspondingforeign language learning target sentences or through comparison withprevious recording files generated by the learner and corresponding tothe learning target sentences, as assistant data in comparison with datafor speech recognition stored in the storage device by the speechrecognition system, or as data for improvement in speech recognitionefficiency through one of the above methods or through a combinationthereof.

According to the present invention, the learner may be informed of adifference between texts generated by incorrectly speaking learningtarget sentences and the learning target sentences among texts generatedby the speech recognition function of the terminal when the learnerspeaks with an intention of correctly speaking the learning targetsentences, and the most incorrect parts may be ranked and the learnermay be informed thereof, thereby guiding the learner to correctincorrectly spoken parts.

According to the present invention, the method may include a process ofdisplaying a message asking to execute foreign language learning afterstoring foreign language content with permission of a copyright holderin a storage device, followed by making it impossible to perform foreignlanguage learning, upon determining that there is no foreign languagecontent with permission of the copyright holder in the storage device,thereby preventing occurrence of illegal matters, such as violation ofcopyright law, by the learner.

According to the present invention, the terminal may provide a pluralityof subtitles in different languages to allow learners of differentlanguages to perform foreign language learning using apps or devicesadopting the method according to the present invention, and may displaysubtitles in a language other than a learner's native language on ascreen of the terminal by learner selection such that the learner canlearn a second foreign language through interactive conversation basedon the subtitles using the TTS function and the speech recognitionfunction.

DESCRIPTION OF DRAWINGS

FIG. 1 is a flowchart of a command recognition process in aconversation-based foreign language learning method using a speechrecognition function and a TTS function according to the presentinvention.

FIG. 2 is a flowchart of a learning mode of a foreign language learningprocess through speech input of a learner in the conversation-basedforeign language learning method according to the present invention.

FIG. 3 is a flowchart of a conversation mode of the foreign languagelearning process through speech input of the learner in theconversation-based foreign language learning method according to thepresent invention.

BEST MODE

The present invention relates to a conversation-based foreign languagelearning method that enables foreign language learning throughinteractive conversation between a learner and a terminal, such as asmartphone, a tablet computer, a notebook computer, a talking toy, an AIspeaker, and the like, based on foreign language content, such asmovies, dramas, pop songs, news, and the like, using a speechrecognition function and a text-to-speech (TTS) function of theterminal.

The conversation-based foreign language learning method using the speechrecognition function and the TTS function of a terminal according to oneembodiment of the present invention may be realized by installing appsadopting the foreign language learning method according to the presentinvention on a portable terminal, such as a smartphone, a tabletcomputer, a notebook computer, and the like, which has the speechrecognition function and the TTS function. This embodiment may beimplemented through Android Studio using Java as an App developmenttool. However, it should be understood that the present invention may beimplemented using various development tools other than Android Studio orusing other computer languages, and an operating system for driving asmartphone is not limited to Google Android and may adopt other systems,such as an operating system of Apple iPhone.

A process of comparing the content of the speech input by the learnerwith the content of a previously stored command and with learning targetcontent may be carried out by at least one selected from the groupconsisting of a method of comparing recording data of learner's speechwith speech data of the learning target content, a method of comparingrecording data of the learner's speech with data possessed by a speechrecognition system, a method of comparing recording data of thelearner's speech with other recording data of the learner's speechstored in advance, a method for comparing text generated by the speechrecognition system with text of the learning target content, and acombination thereof.

The foreign language content may include movies, dramas, pop songs,news, and the like, which include foreign language expressions,subtitles of the foreign language expressions, and subtitles oftranslated native language expressions corresponding to the foreignlanguage expressions. Since there is no limitation on a method ofextracting and storing such basic data, all available methods may beused to achieve the objectives of the present invention. Examples ofsuch storage methods include a method of storing the foreign languagecontent in a specific folder of the terminal to retrieve and execute theforeign language content, a method of downloading content stored in astorage device outside the terminal, such as the cloud, a method ofdownloading the foreign language content from a server on the Internet,and the like through streaming. For specific content, a method ofcreating a database for storage of the specific content to classify andstore data so as to be displayed and used as needed, a method of storingdata, such as foreign language texts, native language texts, commands,time information of subtitles, learning grades, and the like in the formof files in a program, such as a text switcher, HashMap, sharedpreference, and the like without creating the database may be suitablyused. In addition, a method of loading and executing all or part of theforeign language content, particularly, each of learning targetsentences, from the storage device may also be used according toperformance or characteristics of a video or audio player used fornatural conversation-based foreign language learning, which is theobjective of the present invention. For example, the learner maymanipulate the terminal such that a screen or speech for a specificsentence can be realized at a specific playback point (a point where alearning target sentence is present) of the corresponding content, and,if it is difficult to play the corresponding content from the start tothe end due to difference in performance of a corresponding media playeror the terminal, the content may be divided into units of each learningtarget sentence, saved, and then executed. Since each of these methodshas advantages and disadvantages, these methods may be suitably selectedand implemented according to conditions, such as the size of the contentprovided as a learning target, storage capacity and performance of theterminal carried out by the learner, and the like.

In the method according to the present invention, all or some of foreignlanguage sentences in foreign language subtitles of a foreign languagevideo may be input as an array of basic sentences displayed in a textswitcher (String array) and are saved in a program using a text switcherfunction supported by Android Studio and Java; the content of nativelanguage subtitles corresponding to the foreign language sentences arestored corresponding to IDs indicating the sequence number of the basicsentences in the text switcher and keys of the HashMap (a technique forstoring and using data in the form of keys and corresponding values) ofJAVA and an Android Studio program; and all or one of video or audiofiles (here, video files refer to files that execute both images andaudio, and the audio files refer to files that execute only audio. Thevideo or audio file is used according to learner selection) includingthe basic sentences are/is divided according to the correspondingsentence(s) or by chapter(s) set according to the content of thesentence(s) and are/is stored with appropriate title(s) (in the case ofdividing each file according to the corresponding sentence, it isconvenient to attach a title to each file corresponding to the number ofthe sentence) so as to be executed corresponding to a learning targetsentence or a chapter. Here, when execution of an audio file that isdivided and saved for each sentence is carried out using a sound poolfunction (a technique for uploading relatively short audio files in amemory and executing the audio files) of the Android Studio program andJAVA, it is very effective since the audio file of the sentences can becorrectly played unlike a media player. This structure can realize anapplication program that is simple and easy to manage while allowingsuitable implementation of the present invention that is aimed atenabling the learner to perform foreign language learning throughnatural conversation with the terminal via speech transmission. However,it should be understood that the present invention is not limitedthereto and various modification and changes can be created withoutdeparting the scope of the present invention.

Hereinafter, a conversation-based foreign language learning method usinga speech recognition function and a TTS function according to oneembodiment of the present invention will be described with reference toFIG. 1 to FIG. 3.

The conversation-based foreign language learning method using a speechrecognition function and a TTS function according to the embodiment ofthe present invention is a conversation-based foreign language learningmethod through reciprocal speech transmission using foreign languagecontent containing a plurality of foreign language learning targetsentences. Referring to FIG. 1, in the conversation-based foreignlanguage learning method, first, voice commands that can be transmittedto a terminal by a learner are previously stored in the terminal (StepS11). Here, the voice commands may be stored in a server or in the cloudbesides the terminal. When an app is executed in the terminal to displaya screen, the terminal enters a speech waiting state to receive speechinput by the learner and informs the learner of the speech waiting statethrough a sound signal or by displaying the speech waiting state on thescreen together with the sound signal (Step S12)). When the learnerinputs speech in the speech waiting state in which the terminal canreceive the speech input by the learner, the terminal recognizes thecontent of the speech using a speech recognition function of theterminal (Step S13), compares the content of the speech with apreviously stored command (Step S14), executes a corresponding commandupon determining that the content of the speech is coincident with acertain command (Step S15), and returns to the speech waiting state(Step S16). In the comparison step (Step S14), the terminal recognizesthat the speech input by the learner corresponds to an expression withinthe foreign language content upon determining that the speech input bythe learner in the speech waiting state does not match a certaincommand.

Upon determining that the speech input by the learner corresponds to theexpression within the foreign language content, it is determined whetherthe speech input by the learner is the same as a current learning targetsentence or belongs to the same category as the current learning targetsentence (Step S21), as shown in FIG. 2. Upon determining that thespeech input by the learner is the same as the current learning targetsentence or belongs to the same category as the current learning targetsentence (Step S22), the terminal proceeds to the next learning targetsentence (Step S23), provides a sound of the foreign language content ora TTS sound corresponding to the next learning target sentence (StepS24), and enters the speech waiting state again to display the speechwaiting state on the screen and/or to inform the learner of the speechwaiting state through a sound signal (Step S25). Upon determining thatthe speech input by the learner is not the same as the current learningtarget sentence or does not belong to the same category as the currentlearning target sentence (Step S22), the terminal provides the sound ofthe foreign language content or the TTS sound for the corresponding textsentence until it is determined that the speech input by the learner isthe same as the current learning target sentence or belongs to the samecategory as the current learning target sentence (Step S26), and entersthe speech waiting state again to allow the learner to speak again (StepS27) (such a foreign language learning method will hereinafter bereferred to as a foreign language learning mode). Here, when the currentlearning target sentence is repeated due to failure of the learner tocontinuously input speech that belongs to the same category as thecurrent learning target sentence or when the learner inputs simplespeech, such as “yes” and “okay”, or a preset voice command, such as“you speak the next learning target sentence” and the like (it should beunderstood that, since the above example is an example where English isthe target language, different expressions are used for otherlanguages.) (Step S28), the terminal may stop foreign language learningfor the current learning target sentence and may proceed to the nextlearning target sentence (Step S23).

Next, upon determining that the speech input by the learner does notmatch the command and corresponds to an expression for foreign languagelearning (Step S14), it is determined whether the speech input by thelearner is the same as the next learning target sentence of the currentlearning target sentence or belongs to the same category as the nextlearning target sentence (Step S35), as shown in FIG. 3. Upondetermining that the speech input by the learner is the same as the nextlearning target sentence of the current learning target sentence orbelongs to the same category as the next learning target sentence, theterminal provides a sound of the foreign language content or a TTS soundcorresponding to the next learning target sentence of the correspondingforeign language learning target sentence, that is, a sentence next tothe next learning target sentence of the current learning targetsentence (Step S36), and informs the learner of the speech waiting stateto allow the learner to speak the next learning target sentence (StepS37) (Such a foreign language learning method will hereinafter bereferred to as a conversation mode).

On the other hand, when preparing to execute all or some of the foreignlanguage content for each of learning target sentences or for each ofdivided chapters, detailed processes are performed to allow the learnerto perform foreign language learning through natural conversation withthe terminal using such foreign language content. Here, unlike a typicalforeign language learning method using general foreign language contentin which the learner watches video while pressing buttons on the screenone by one, the foreign language learning method according to thepresent invention is aimed at enabling conversation-based foreignlanguage learning by allowing the learner to input speech to theterminal such that the terminal properly responds to the speech input bythe learner. The detailed processes for achieving this purpose will bedescribed in the sequence of the problems summarized above as follows.

(1) Detailed Description and Examples of a Solution to the First Problem(Allowing Conversation-Based Foreign Language Learning Through SpeechTransmission Instead of Contact)

In the conversation-based foreign language learning method throughreciprocal speech transmission using the speech recognition function andthe TTS function of the terminal according to the present invention,most procedures from the beginning to a completion time of operation ofthe terminal are carried out through reciprocal speech transmissionbetween the learner and the terminal while minimizing contact betweenthe learner and a button of the terminal or, if any, a screen of theterminal. Here, it should be noted that the conversation-based foreignlanguage learning method according to the present invention does notrequire the learner to perform the entire procedure of foreign languagelearning only through speech transmission without any manipulation ofthe terminal by the learner to operate the terminal through contact withthe screen of the terminal or by pressing the button of the terminal(hereinafter referred to as a contact method) instead ofconversation-based foreign language learning through speech transmission(hereinafter referred to as a speech transmission method) foreignlanguage learning. This is because there can occur a special situationwhere the terminal is turned off or the speech waiting state does notappear for technical reasons in the course of foreign language learningusing a terminal including a screen or where the contact method is moreadvantageous for natural conversation than the speech transmissionmethod and thus is inevitably used together with the speech transmissionmethod. In the method according to the present invention, most mainprocedures are performed by the speech transmission method. Thus, itshould be understood that even an implementation using the contactmethod together with the speech transmission method with an intention ofevading the scope of the present invention can also be interpreted asfalling within the scope of the present invention in practice if asubstantial part of the main procedure is carried out based on thespeech transmission method. In one embodiment of the present invention,as soon as the terminal starts to execute an app to output a soundcorresponding to the first learning target sentence or to display thefirst learning target sentence together with the sound, the terminalinforms the learner of a speech waiting state through a sound signal orby displaying the speech waiting state together with the sound signalsuch that the learner can input speech into the terminal. Such afunction may be implemented by including a command to show the speechwaiting state in the content to be executed upon start of a program foroperating the terminal, such as the onStart function of Java used inAndroid Studio, without being limited thereto. In this state, thelearner may start foreign language learning by transmitting foreignlanguage speech to the terminal using a predefined command or an exactforeign language expression in the stored content. In some embodiments,the terminal may send a sound signal to inform the learner of the speechwaiting state with respect to the first learning target sentence or theterminal including a screen may display the learning target sentence onthe screen together with the sound signal such that the learner canwatch the learning target sentence and speaks the learning targetsentence. In addition, the terminal may read the first learning targetsentence to the learner through a content sound or a TTS sound or mayprovide a welcome greeting and guidance for the foreign languagelearning method to the learner through a preset content sound or TTSsound depending on settings. In some embodiments, the terminal may storea location where the learner finished foreign language learning beforeand may move to the location upon restart of the app to read thecorresponding foreign language learning target sentence to the learnerwith a content sound or a TTS sound and to display the speech waitingstate in which the learner speaks after the terminal.

In the speech waiting state where a program adopting the methodaccording to the present invention is executed in the terminal, thelearner performs foreign language learning for a current sentence readby the terminal or speaks a command to the terminal to move to anotherchapter or the next learning target sentence at which the learner wantsto perform foreign language learning. When the current sentence is asentence that the learner wants to learn, the learner may start newlearning by speaking after the terminal reads the sentence. With respectto the speech input by the learner, the terminal reacts according to adetermination result as to whether the speech input by the learnercorresponds to a command or is the same as any one of learning targetsentences or belongs to the same category as any one of the learningtarget sentences and such reaction includes display of the speechwaiting state at the end of reaction such that the learner can inputsnext speech. As such, in the foreign language learning method accordingto the present invention in which the learner inputs speech to theterminal through speech transmission in the speech waiting state of theterminal and the terminal reacts with respect to the speech input by thelearner, many procedures of foreign language learning may be carried outthrough speech transmission between the learner and the terminal insteadof the contact method, thereby enabling natural conversation-basedforeign language learning through reciprocal speech transmission betweenthe learner and the terminal.

In order to allow continuous natural conversation-based foreign languagelearning through reciprocal speech transmission between the terminal andthe learner without interruption, it is necessary to prevent the speechwaiting state from being lost and the present invention uses thefollowing method to this end. In order to prevent the speech waitingstate of the terminal from being easily lost, when the learnerrecognizes the speech waiting state of the terminal and speaks anyexpression that is recognized as a foreign language, the terminaldetermines that the current sentence is incorrectly expressed by thelearner, reads a correct sentence of the current sentence to the learnerwith a content voice or a TTS sound, and then enter the speech waitingstate again to provide a sound signal such that the learner canrecognize the sound signal without watching, if any, the screen of theterminal. As a result, when the learner recognizes that the terminal isin the speech waiting state, since the speech waiting state of theterminal can be prevented from being lost by allowing the learner tospeak the simplest foreign language expression (but longer than theminimum length required by a speech recognition program), such as asingle word in the current sentence, it is possible for the learner tomaintain the conversation state through speech input to an extent thatthe learner wants to continue conversation with the terminal. Inaddition, even though the learner should move on to the next learningtarget sentence after completing learning of the current sentence, ifthe learner continues to fail to speak the current sentence similarlyover a predetermined degree, there can be a problem that the learner isrequired to repeat the current sentence only. Even in this case, whenEnglish is a learning target language, the method according to thepresent invention allows the learner to proceed to learning of the nextlearning target sentence by allowing the learner to speak a simple word,such as “yes” and “okay”, which is easy for the terminal to recognize,or to speak a predetermined command, such as “you speak the nextlearning target sentence” and the like. By these means, since thelearner can prevent the speech waiting state of the terminal from beinglost, it is possible to prevent interruption of naturalconversation-based learning.

(2) Detailed Description and Examples of a Solution to the SecondProblem (Preparation of Learner Command for Speech Transmission)

In order to achieve the purpose of the present invention to enableforeign language learning through conversation between the learner andthe terminal, voice commands to be transmitted to the terminal by thelearner are prepared and stored in the terminal such that the terminalallows conversation-based foreign language learning through reactionwith respect to the content of the command when the learner delivers acertain command with voice. Reaction of the terminal with respect tospeech transmission of the learner is carried out by a particular methodso as to allow natural conversation-based foreign language learningthrough reciprocal speech transmission. That is, upon receiving thespeech input by the learner, the terminal executes a correspondingoperation upon determining that the command is coincident with any oneof the preset commands. Such commands include a movement command to moveforwards or backwards by a single or plural sentences between learningtarget sentences or to move to a previous chapter or the next chapter ofa current chapter or to a particular chapter when the learning targetsentences are divided by chapters, an execution command, an executioncommand to read a current sentence or a corresponding sentence with acontent sound or a TTS voice while sequentially moving through about 10or 20 sentences from the current sentence or while skipping thesentences one by one, a video execution command to provide a screen ofthe terminal together with or without subtitles for the content of eachchapter for the terminal including the screen, an audio executioncommand to play a recorded audio file of corresponding content, asubtitle display command to display subtitles upon execution of video oraudio content when the terminal includes the screen, a follow-speechcommand to provide a time for the learner to speak a current sentenceafter the terminal and to read the next learning target sentence afterexecution of a content sound or a TTS sound for the current sentence isfinished, a role play command to read a predetermined number ofsentences, for example 10 or 20 sentences, or all sentences of theentire chapter to the learner such that the learner can speak after theterminal in a role-play manner, a command to read corresponding nativelanguage subtitles with a TTS sound for a hint or explanation for acurrent sentence or the next learning target sentence that the learnerwill speak, a command to provide a list of preset voice commands in textor sound format, a command to finish learning and close the app, and thelike. In addition, in response to such various movement commands, theterminal enters the speech waiting state after proceeding to acorresponding sentence and reading the corresponding sentence with acontent sound or a TTS sound, thereby enabling conversation-basedforeign language learning through reciprocal speech transmissionaccording to the present invention. Examples of such commands are asfollows.

“Please show me the voice command list” (terminal providing voicecommand list),

“Yes” (terminal reading next learning target sentence to learner withcontent sound),

“Okay” (terminal reading next learning target sentence to learner withTTS sound),

“Please speak this sentence” (terminal reading current sentence tolearner with content sound),

“Please speak the next learning target sentence” (terminal reading nextlearning target sentence to learner with content sound),

“Please let me hear episode speaking” (terminal reading a predeterminednumber of sentences to learner with TTS sound),

“Please let me hear original sound episode speaking” (terminal readingpredetermined number of sentences to learner with content sound),

“Please let me follow your speaking” (terminal reading a predeterminednumber of sentences to learner one by one with TTS sound and providingtime to allow learner to speak each sentence after terminal),

“Please let me follow your speaking 100 times” (terminal reading 100sentences to learner one by one with TTS sound and providing time toallow learner to speak each sentence after terminal),

“Please let me follow your original sound speaking” (terminal reading apredetermined number of sentences to learner one by one with contentsound and providing time to allow learner to speak each sentence afterterminal),

“Please let me follow your original sound speaking 100 times” (terminalreading 100 sentences to learner one by one with content sound andproviding time to allow learner to speak each sentence after terminal),

“Please skip and play together” (terminal and learner alternatelyreading a predetermined number of sentences one by one with TTS sound),

“Please skip and play original sound speaking together” (terminal andlearner alternately reading a predetermined number of sentences one byone with content sound),

“Please repeat this sentence” (terminal repeatedly reading currentsentence to learner a predetermined number of times with TTS sound),

“Please repeat original sound this sentence” (terminal repeatedlyreading current sentence to learner a predetermined number of times withcontent sound),

“Hint” (terminal providing native language subtitles of next learningtarget sentence),

“Wait” (terminal stopping speech waiting state for temporary stopping oflearning),

“Music”, “Play music”, “Would you play a music for me” (terminalreplaying audio file of content of corresponding chapter),

“Movie”, “Play movie”, “Would you play a movie for me” (terminalreplaying video file of content of corresponding chapter),

“Next chapter”, “Please go to next chapter”, “Would you please bring meto next chapter” (terminal jumping to next chapter),

“Previous chapter”, “Please skip to previous chapter”, “Would you skipto previous chapter” (terminal jumping to previous chapter),

“Seventh chapter, chapter” (terminal jumping to particular chapter),

“Next 11, 20, 50, 100, 500, 1000” (terminal jumping forward bynumerals),

“Previous 11, 20, 50, 100, 500, 1000” (terminal jumping backward bynumerals),

“First” (terminal jumping to first sentence), “Last” (terminal jumpingto last sentence),

“Good job see you again soon” (terminal storing learning result grandand finishing learning) [99] Such commands enable naturalconversation-based foreign language learning through speech transmissionbetween the learner and the terminal without the contact method in mostprocedures from the start of the terminal implementing the presentinvention to the last moment of an operation executed by the terminal.

The method according to the present invention is characterized byenabling natural conversation-based foreign language learning based onreciprocal speech transmission between the terminal and the learnerwhile minimizing contact between the learner and a button or screen ofthe terminal. Thus, according to the present invention, most commands tobe delivered to the terminal by the learner are previously stored in theform of voice command files in the terminal such that the terminal canexecute an operation corresponding to a command in response to thecommand from the learner. However, as the number of such voice commandsincreases, it can become increasingly difficult to distinguish whetherthe learner speaks a voice command or any one of learning targetsentences. Accordingly, an identifier for making it clear that thecontent of a voice expression delivered by the learner is not a learningtarget sentence but a voice command to be performed by the terminal canprevent occurrence of such confusion. For example, foreign languagevideo content, such as movies, dramas, or news, may be divided intochapters each having a suitable length for learners to learn accordingto the content of a scene or a plot thereof so as to help the learnersunderstand and remember the foreign language video content by watchingor listening to the corresponding video or audio in each chapter andlearning sentences in each chapter as learning target sentences. In thisway, if the foreign language video content are divided into chapters andthe divided chapters are given appropriate titles according to thecontent of the chapters, it will be very helpful for the learner toremember the content of the corresponding chapters and to learn thesentences in the chapter again or to refresh memory of the sentences inthe chapter. Since the title of each chapter may be used for a movementcommand to move to a corresponding chapter, the command using the titleof the chapter can cause confusion with the learning target sentence ofthe foreign language video content. According to the present invention,an identifier created using an expression that does not generally appearin the learning target sentence is used after the title of the chapterto prevent confusion between the voice command and the learning targetsentence. For example, when English is a learning target language, anexpression that does not generally appear in dialogues of foreignlanguage video content, such as “chapter”, is added as an identifierafter the title of the chapter. For example, for movement of theterminal to a chapter having a title of “looking for a motive”, when thelearner speaks an identifier “chapter” after the title of the chapter,that is, “looking for a motive”, the terminal recognizes this command asa command to skip to the corresponding chapter and execute thecorresponding command. Use of such an identifier is not limited to amovement command between chapters, but may also be used for other voicecommands, such as a command to play video or audio of sentences within achapter and the like.

(3) Third Problem (Implementation of Learning Method in a Learning Mode)

Through continuous appearance of the speech waiting state and speechtransmission based on various commands delivered from the learner to theterminal learner, the learner can repeat learning by listening to thecontent sound or the TTS sound from the terminal for a current learningtarget sentence and speaking after the terminal. In such a learningprocess, the learner may select to make the terminal read the learningtarget sentence with either the content sound or the TTS sound. Thecontent sound that gives accurate pronunciation is necessary forlistening training and is very effective for repeated training such thatthe learner can speak a foreign language with accurate pronunciation. Inthe case where the terminal does not include a screen or it is difficultto understand the content sound at night or due to noise, the TTS soundcan be more effective since the TTS sound is clearer than the contentsound. The learner may repeat learning of learning target sentences bycombining the aforementioned various commands in various ways whilechanging the terminal's sound to the content sound or the TTS soundaccording to the situation. However, depending on learner's ability anddifficulty of the foreign language content, it is not easy for thelearner to speak a foreign language sentence to the speech recognitionsystem of the terminal to exactly same degree as the foreign languagesentence in the foreign language content. In this case, such a problemcan be solved by grading the degree of similarity in determination ofcorrectness of an expression created by the learner such that theterminal responds differently according to the similarity grade insteadof guiding the learner to continue to speak the same sentence until thelearner makes an exactly correct expression if the learner fails to makea correct expression. That is, the degree of similarity is graded indetermination of the degree of similarity between what the learnerspeaks and a sentence to be compared (including a current learningtarget sentence and the next learning target sentence). Then, upondetermining that an expression created by the learner is higher than apreset grade (that the expression created by the learner belongs to thesame category as the learning target sentence), the terminal reads thenext learning target sentence to the learner and enters the speechwaiting state by determining that the learner carries out learning ofthe learning target sentence to a predetermined level of learning. Onthe contrary, upon determining that the expression made by the learneris less than a preset grade, the terminal reads the current learningtarget sentence to the learner and enters the speech waiting state toguide the learner to speak again after the terminal by determining thatthe learner fails to made an expression belonging to the same categoryas the learning target sentence. Through such a process, the methodaccording to the present invention allows the learner to performconversation-based foreign language learning while naturally improvingthe level without giving up. By way of example, when speech input by thelearner is correctly the same as a comparison target sentence, thespeech may be rated as Level A; although the speech input by the learnerfails to achieve Level A and is not correctly the same as the comparisontarget sentence, when the speech input by the learner is substantiallysimilar to the comparison target sentence to an extent that the learneris allowed to speak the next sentence, it may be rated as Level B(belonging to the same category as the learning target sentence),thereby allowing the terminal to move to the next sentence even thoughthe learner fails to achieve Level A; and when the speech input by thelearner is below B level, it may be rated as Level C.

(4) Detailed Description and Examples of a Solution to the FourthProblem (Implementation of Interactive Natural Conversation)

A learner who has learned to a certain extent by the above methods canhave an ability to speak a certain sentence in the foreign languagecontent without listening to a sound from the terminal. In this case,the learner may require a method of learning a foreign language in aconversation mode in which the learner has a reciprocal conversationwith the terminal, instead of continue learning in the above learningmode, such as follow speaking and the like. In order to enable both thelearning mode and the conversation mode without deviating from theobjective of the present invention to provide a naturalconversation-based foreign language learning method through speechtransmission rather than the contact method, the following method may beused.

That is, if the speech input by the learner is not a command, it isdetermined whether the speech input by the learner is the same as thenext sentence of a current learning target sentence or belongs to thesame category as the current learning target sentence. It is consideredthat the learner wants to proceed with learning in the conversationalmode from a sentence that the learner has spoken instead of learning inthe learning mode for the current sentence, upon determining that thespeech input by the learner is the same as the next sentence of thecurrent learning target sentence or belongs to the same category as thecurrent learning target sentence. Thus, the terminal proceeds to thenext sentence of the sentence spoken by the learner, that is, the nextsentence of the current learning target sentence, reads the nextsentence to the learner with the content sound or the TTS sound, andenters the speech waiting state.

With this method, the learner can perform foreign language learning inthe learning and in the conversation mode without using a particularmethod for change between the learning mode and the conversation mode,and can perform natural conversation-based foreign language learning inthe conversation mode even without speaking an exactly correctexpression as in the learning mode.

Next, operation of the terminal in the learning mode and theconversation mode will be described in detail. Upon determining that thespeech input by the learner is the same as the next learning targetsentence of a current learning target sentence or belongs to the samecategory as the next learning target sentence thereof, the terminaldetermines that the learner wants to learn the next learning targetsentence of the current learning target sentence in the conversationmode instead of learning the current learning target sentence read bythe terminal. That is, upon determining that the learner inputs the samespeech as the next learning target sentence (for example, Sentence No.8) of the current sentence (for example, Sentence No. 7) or speechbelonging to the same category as the next sentence instead of speakingthe current sentence, the terminal reads a sentence (Sentence No. 9)after the next sentence of the current sentence to the learner andenters the speech waiting state. As such, even after the learner inputsthe same speech as the next sentence (Sentence No. 8) or a speechbelonging to the same category as the next sentence (Sentence No. 8)instead of speaking the current sentence (Sentence No. 7), and theterminal proceeds to a sentence after the next sentence of the currentsentence, that is, Sentence No. 9, to read Sentence No. 9 to the learnerand enters the speech waiting state to operate in the conversation mode,the learner may perform foreign language learning in the learning modeby reading the previous sentence (Sentence No. 9) after the terminal ormay try to enter the conversation mode by reading the next sentence(Sentence No. 10) of the previous sentence read by the terminal even inthe case where the terminal reads the previous sentence (Sentence No. 9)again to the learner due to failure of the learner to input the samespeech as the next sentence (Sentence No. 10) or a speech belonging tothe same category as the next sentence thereof.

According to the present invention, when the speech input by the learneris neither a voice command nor the same as the current sentence or thenext learning target sentence, or belongs to the same category as thecurrent sentence or the next learning target sentence, the foreignlanguage learning method may further include determining whether thespeech input by the learner is the same as any one sentence among theentire sentences other than the current sentence or the next learningtarget sentence. When the speech input by the learner is the same as anyone sentence among the entire sentences other than the current sentenceor the next learning target sentence, the terminal determines that thelearner wants to start learning from the corresponding sentence in thelearning mode or in the conversation mode. Then, the terminal proceedsto the next learning target sentence of the sentence spoken by thelearner and reads the next learning target sentence to the learner witha content sound or a TTS sound and then enters the speech waiting state.

(5) Detailed Description and Examples of a Solution to the Fifth Problem(Calculation of Suitable Time for Terminal to Enter Speech WaitingState)

According to the present invention, a period of time until the terminalenters the speech waiting state may be set in consideration of a lengthof a corresponding foreign language learning target sentence and a timethat the corresponding foreign language learning target sentence isexecuted in the foreign language content.

According to the present invention, in response to speech input by thelearner, the terminal executes an operation corresponding to the contentof each speech and enters the speech waiting state to allow the learnerto continue learning in the conversation mode. Here, when a period oftime until the terminal enters the speech waiting state is suitablyadjusted, it is possible to allow the learner to continue learning in amore natural conversation mode. Such a suitable period of time until theterminal enters the speech waiting state is set in consideration of atime for the terminal to execute a corresponding command or the currentor next sentence (hereinafter referred to as a terminal consuming time)and a time for the learner to speak after the terminal (hereinafterreferred to as a learner consuming time) in the case where the methodincludes a process where the learner speaks after the terminal (aprocess where the learner performs foreign language learning by simplyspeaking after the terminal, for example, a follow-speech process,instead of speaking after the terminal enters the speech waiting state).In some cases, the period of time until the terminal enters the speechwaiting state is determined without considering the terminal consumptiontime, for example, in the case where the terminal is simply allowed toenter the speech waiting state after finishing overall play of all videoor audio files or play of the video or audio files chapter by chapter.In these cases, the period of time until the terminal enters the speechwaiting state is determined simply in consideration of the learnerconsumption time. Here, the terminal consumption time differs dependingupon a method in which the terminal reads a current sentence to thelearner. When the terminal reads the current sentence with the contentsound, the period of time until the terminal enters the speech waitingstate is determined depending upon how fast the corresponding sentenceis executed in the foreign language content. On the other hand, a timefor which the terminal reads the current sentence with the TTS sound isproportional to the number of characters in the corresponding sentence.In general, the time for which the terminal reads a sentence with acontent sound is longer than the time for which the terminal reads thesentence with a TTS sound since, for example, since a time for anexecution device, such as a media player, to prepare for play and reactin order to play the content sound is longer than a play time of the TTSsound. A suitable period of time until the terminal enters the speechwaiting state is set in consideration of this point. A suitable terminalconsumption time or learner consumption time is set in consideration ofa length of the learning target sentence and a time for which thecorresponding sentence is executed in the foreign language content.Although an execution time with the content sound is generallydetermined after calculating a time for displaying subtitles based ontime information contained in the subtitles and an execution time withthe TTS sound is generally determined in proportion to the number ofcharacters in the corresponding sentence, the period of time until theterminal enters the speech waiting state may be set in consideration ofboth the length of the corresponding sentence length and an executiontime thereof or only one of the sentence length and the execution time.When the sentence length and the execution time are calculated, thecalculation result may be applied as it is and the sentence length orthe execution time may be divided into several stages to set a differenttime for each stage. More accurate calculation of the period of timeuntil the terminal enters the speech waiting state enables more naturalconversation-based learning. The sentence length can be extracted usinga function that calculates the number of characters in a sentence (forexample, length( ), and the like, and the execution time may beextracted from a video subtitle file of or an audio lyrics file. Sincethe subtitle or lyric file contains information indicating a time forwhich each subtitle or lyric is displayed on the screen and includesstart and end times of the subtitles for each subtitle unit (when noseparate ending time is indicated, a start time of the next subtitlebecomes an end time of a previous subtitle), the execution time of thesentence may be calculated by subtracting the start time from the endtime and may be displayed together with the corresponding subtitles.

By way of example, assuming the execution time of the current sentencein the subtitle file is 3 seconds and the terminal reads a certainsentence to the learner with a content sound once in response to speechinput by the learner and enters the speech waiting state after thelearner speaks the same sentence once after the terminal, the minimumperiod of time until the terminal enters the next speech waiting stateafter execution of the terminal is finished in response to the speechinput by the learner becomes 6 seconds by adding 3 seconds of theterminal consumption time to 3 seconds of the learner consumption time.Here, since it is necessary to provide an appropriate additional timeaccording to performance of the terminal, characteristics of the mediaplayer using the terminal, and learner's ability to listen to and speaka foreign language, a finally calculated suitable time may be 7 to 8seconds. Such an additional time may be suitably set according to eachembodiment. In addition, when the terminal executes the TTS soundinstead of the content sound, the execution time of the terminal may beset according to the length of the sentence.

(6) Detailed Description and Examples of a Solution to the Sixth Problem(Correction for Matching Both Sentences)

According to embodiments of the present invention, it is determinedwhether the speech input by the learner is the same as a comparisontarget sentence or belongs to the same category as the comparison targetsentence by converting the speech input by the learner in the speechwaiting state into text through recognition of a speech recognition unitin the terminal, followed by comparing the text with the text of thecomparison target sentence. Here, in comparison of the text generatedthrough recognition of the speech with the text of the comparison targetsentence, the foreign language learning method may further includecorrection of both texts or one of the texts in order to allow morenatural conversation-based learning. That is, since there can bediscordance between the sentence in the subtitles of the foreignlanguage content and the text generated from the speech input by thelearner by a speech recognition program, such discordance often causesfailure in correct determination even though the speech input by thelearner is exactly the same as the current sentence. For example, suchincorrect determination as to similarity between the text and thesentence in the subtitles is caused by differences caused by expressingsubstantially the same meaning using different expression methodsdepending on whether sentences to be compared are case-sensitively used,whether punctuation marks or special characters are used, how to expressnumbers, whether abbreviations are used, and the like. Accordingly, theforeign language learning method according to the present invention mayfurther include the step of correction for actual comparison of bothcharacter strings or with one character string so as to determine actualsimilarity between sentences. The correction step includes removingunnecessary blanks or treating the unnecessary blanks as being absent incomparison of the text with the sentence in the subtitles in order toprevent the number and positions of blanks in both sentences fromaffecting a comparison result. In the correction step, the texts of bothsentences may be compared assuming that punctuation marks, such asquestion mark (?), exclamation point (!), comma (,), period (.),quotation mark (“ ”), hyphen (-), center point (⋅), parentheses {( )},curly braces ({ }), square brackets ([ ]), or spaces incorrectly addedat the beginning and end of sentences, and spaces overlapping each otherare absent; and that an expression of numbers with Arabic numerals andan expression of numbers with letters; an expression of following verbsor auxiliary verbs in abbreviated forms using an apostrophe (') after asubject and an expression of original forms thereof without usingabbreviations; an expression of negative sentences in abbreviated formsusing an apostrophe (') and an expression of original forms thereofwithout using abbreviations; “going to” and “gonna”; “want to” and“wanna”; “got to” and “gotta”; or a sentence used by distinguishingbetween uppercase and lowercase characters and a sentence used withoutdistinction therebetween may be regarded as being exactly or partiallyidentical to each other.

For example, correction may be realized using a function that changesuppercase characters to lowercase characters (for example, lowerCase( ),a function that replaces specific characters with other characters (forexample, replace( ), and the like to correct the sentences generatedfrom subtitles or the text generated from the speech input by thelearner to be coincident with each other. Through the correction step,more accurate comparison between the sentences (when two sentences havesubstantially the same meaning and the content of sounds correspondingto the two sentences is the same, the two sentences are treated as thesame) can be achieved, thereby enabling more natural conversation-basedforeign language learning.

(7) Detailed Description and Examples of a Solution to the SeventhProblem (Additional Means for Facilitating InteractiveConversation-Based Learning Between Learner and Terminal)

Upon foreign language learning in the conversation mode described above,it is not easy for some learners to memorize sentences that the learnerwill speak without following the terminal after listening to sounds ofthe terminal. To compensate for this problem, the terminal including thescreen is operated to allow a current foreign language sentence to bedisplayed together with the next foreign language sentence, whereby thelearner can easily try to start foreign language learning in theconversation mode. As a result, the learner can watch the currentlearning target sentence together with the next learning target sentenceon the screen. Thus, when the learner wants to perform learning of thecurrent learning target sentence in the conversation mode after stoppinglearning of the current learning target sentence in the learning mode,the learner may try to enter foreign language learning in theconversation mode by speaking the next learning target sentence insteadof speaking the current sentence after the terminal. After the learnerperforms foreign language learning to some extent, since it is veryimportant for the learner to continue learning through reciprocal speechtransmission together with the terminal, it is meaningful to provide ameans for displaying the next learning target sentence. Since thecurrent learning target sentence may be displayed together with the nextlearning target sentence on the terminal by a typical method known inthe art, detailed description of the method will be omitted.

In some cases, depending on learning ability of the learner andcharacteristics of a learning target foreign language, effectivelearning cannot proceed since it is too difficult for the learner tolisten to the entire sentence as a learning target from the beginning ofthe sentence and to guide the learner to make an expression after theterminal. In this case, in order to facilitate foreign language learningthrough conversation between the learner and the terminal, the terminalmay read only a part of a specific sentence, for example, apredetermined number of words or characters, among learning targetsentences to the learner through the TTS function, instead of readingthe entirety of the specific sentence, enter the speech waiting state toallow the learner to speak after the terminal, and determines whetherspeech input by the learner is the same as the part of the specificsentence or belongs to the same category as the part of the specificsentence. Then, the terminal proceeds to another part of the specificsentence to be learned by the learner upon determining that the speechinput by the learner is the same as the part of the specific sentence orbelongs to the same category as the part of the specific sentence, orrepeatedly reads the part of the specific sentence to the learner forlearning upon determining that the speech input by the learner is notthe same as the part of the specific sentence or does not belong to thesame category as the part of the specific sentence. In this case, a partof the beginning of each learning target sentence may be used as alearning target or one learning target sentence may be divided into apredetermined number of words or characters to allow the learner toproceed to the next learning target sentence after completing theentirety of the learning target sentence. Here, the part of the learningtarget sentence may be determined depending upon a predetermined numberof words or characters by learner selection and the terminal reads thispart to the learner with a TTS sound and enters the speech waiting stateto allow the learner to speak after the terminal. Then, the terminal mayoperate in response to a speech input by the learner through recognitionof the speech. Details of this operation are the same as the case whereforeign language learning is performed using the entirety of thelearning target sentence and detailed description thereof will beomitted.

Further, as an additional means for facilitating foreign languagelearning through conversation between the learner and the terminal, theterminal may read the content of native language subtitles correspondingto the learning target sentence with a TTS sound before the terminalreads the learning target sentence to the learner with the content soundof the foreign language content or the TTS sound. As a result, thelearner can listen to the content sound of the foreign language contentor the TTS sound after understanding the meaning of the learning targetsentence to be read to the learner by the terminal, thereby facilitatinglearning.

In particular, when the learner cannot quickly read the subtitles in hisor her native language due to visual impairment or lack of readingability or when it is difficult for the learner to understand themeaning of the learning target sentence read to the learner by theterminal due to ambient noise, foreign language learning may be easilyperformed using the above method.

In the method according to the present invention, a learner who alreadyhas considerable foreign language skills may perform foreign languagelearning through listening to the content of subtitles in his or hernative language to guess the corresponding foreign language sentence inhis or her mind in advance, followed by checking whether his or herguess is correct based on sounds of the foreign language content.

Further, as an additional means for facilitating foreign languagelearning through conversation between the learner and the terminal, theterminal may read the learning target sentence to the learner with theTTS sound before reading the corresponding sentence to the learner withthe content sound. In general, since TTS sound tends to be heard clearlycompared to the foreign language content sound, the above means allowsthe learner not only to listen to the learning target sentence moreclearly, but also to have an effect of repeating the same sentence,thereby facilitating foreign language learning.

Further, as an additional means for facilitating foreign languagelearning through conversation between the learner and the terminal, theterminal may read the content of the foreign language learning targetsentence and the content of native language subtitles corresponding tothe learning target sentence to the learner with TTS sounds thereofbefore the terminal reads the learning target sentence to the learnerwith the content sound with respect to the learning target sentence. Asa result, the learner can also clearly understand the meaning of thelearning target sentence and can repeatedly listen to the foreignlanguage content sound again after listening to the learning targetsentence with a clear pronunciation, thereby facilitating foreignlanguage learning.

(8) Detailed Description and Examples of a Solution to the EighthProblem (Recovery Method Upon Disappearance of Speech Waiting State)

As described above, the present invention uses a means for preventingthe speech waiting state from disappearing in order to enable naturalconversation-based learning through reciprocal speech transmission.However, despite the use of such means, there can be cases where thespeech waiting state disappears. In general, the speech recognition unitof the terminal waits for speech input for about 6 to 7 seconds afterthe speech waiting state starts, and the learner is required to input aspeech in a designated language for this period of time (when there isno speech input in this speech waiting state, the speech waiting statedisappears or the terminal enters a locked state). According to thepresent invention, the terminal may be set to accept a learner'soperation of pressing one or more buttons on an external remote control(including a pen of Samsung Galaxy Note) or earphones connected theretothrough an electric line or electronically by a Bluetooth function as acommand for entering the terminal into the speech waiting state suchthat the terminal can enter into the speech waiting state by pressing acorresponding button on the remote controller or the earphones when thespeech waiting state disappears from the terminal, even without pressinga button on the terminal to send the command for turning the terminalinto the speech waiting state while watching the screen of the terminal,if any. Such a remote control may include a remote control that acts asa button of a camera, and a writing pen of a mobile phone that has abutton provided with a function of another app to act as a remotecontrol upon pressing the button, without being limited thereto.Alternatively, a separate remote controller having the same function maybe used. Such a method of entering the terminal into the speech waitingstate may include use of programs, such as Google Assistant and SamsungBixby, besides use of separate hardware, such as a remote control andearphones. Google Assistant can perform certain functions upon input ofa Google Assistant command, such as OK Google, through a voice even whena screen of a mobile phone is locked. Thus, one of such functions may beset to a command for entering the terminal into the speech waiting statein an application adopting the method according to the present inventionor may be set to a button of the terminal for execute the command,whereby the learner can enter the terminal into the speech waiting statewithout touching the screen of the terminal. If it is difficult to enterthe terminal into the speech waiting state, which is provided as aninternal function of the terminal, using such a remote control, it ispossible to execute the above program again using the remote controller.As described above, since the above program automatically enters theterminal into the speech waiting state upon installation thereof, thelearner may finish previous apps by inputting a command to secondlyexecuted apps through speech input to continue conversation for foreignlanguage learning on the newly executed apps. In this case, on the newlyexecuted apps, the learner may perform foreign language learning from apoint where the previous apps ended.

(9) Detailed Description and Examples of a Solution to the Ninth Problem(Calculation and Display of Learning Performance)

The present invention may further include a means for expressinglearning performance of the learner as a numerical value based onevaluation of the learning performance in the course of foreign languagelearning to inspire learner's motivation to learn a foreign languagewhile allowing parents or teachers to confirm achievements of thelearner. Such means are not necessarily used in all cases and may beadditionally selected according to purposes and learning targets of thelearner or learner's parents or teachers. Specifically, when it isevaluated that the speech input by the learner is exactly the same asthe learning target sentence, the highest score among the scores thatcan be given to the corresponding sentence will be given to the learner,and when it is evaluated that, although the learner fails to speakexactly the same sentence, the learner speaks similarly to some extentthat the learner is allowed to learn the next sentence, the scores willbe differently given according to the degree of similarity of the speechinput by the learner to the learning target sentence. When the level ofthe speech input by the learner is less than the level at which thelearner is allowed to learn the next sentence, the terminal requires thelearner to continue to speak the next sentence without imparting a scoreto the speech input by the learner. In addition, even when it isevaluated that the speech input by the learner is exactly the same asthe learning target sentence, the scores may be differently givendepending upon whether the learner inputs the speech while watching thelearning target sentence or without watching the learning targetsentence after completely memorizing the learning target sentence,whether the learner inputs the speech while watching only the nativelylanguage subtitles or through inference of the remainder of the learningtarget sentence based on some part of the learning target sentence givenas a hint while watching some of the foreign language or native languagesubtitles as a hint, or whether the learner inputs the speech in thelearning mode or in the conversation mode. When the learner inputspreset command speech to the terminal, a score may be givencorresponding to difficulty of speaking the preset command. In thiscase, the degree of similarity of the command speech input by thelearner to a preset command stored in the terminal is not separatelydetermined and the score may be given by evaluating that the learnerspeaks the corresponding command only when the command speech input bythe learner is exactly the same as the preset command. According to thepresent invention, for evaluation of learning performance and forinspiration of motivation to learn a foreign language, the learningperformance of the learner may be evaluated to impart a scorecorresponding thereto, and the foreign language learning method mayadditionally employ a point accumulation system that stores the scorewhen the learner finishes use of the terminal according to learner'sselection to accumulate the scores in addition to the previouslyobtained scores upon start of next learning.

(10) Detailed Description and Examples of a Solution to the TenthProblem (Storage of Voice-Recording Data of Learner)

When the terminal proceeds to the next sentence for learning based ondetermination that the speech input by the learner is exactly the sameas the learning target sentence, voice recording data generated byrecording a speech input by speaking a certain learning target sentencewith a voice of the learner may be stored corresponding to the learningtarget sentence to be used as needed. In this method, when the terminalenters the speech waiting state to allow the learner to input speechwith respect to a certain learning target sentence, the speech input bythe learner is recorded by the speech recognition program and is given atitle corresponding to the learning target sentence to be stored in thestorage device instead of using the recording data for speechrecognition. Here, the recording data may be stored after being replacedby new recording data or together with the new recording data accordingto learner selection. The voice recording data of the learner stored inthis method may be used by the learner in various ways, for example, byimplementing conversation with the terminal or the voice recording datawhile listening to the voice recording data or by continuously playingthe voice recording data to correct mistakes in the speech of thelearner while listening to the voice recording data. Such voicerecording data of the learner may be used to compensate for a limitationof the speech recognition program used for implementation of the methodaccording to the present invention. Depending upon performance of thespeech recognition program, learner ability or situations, when thelearner continuously fails to proceed to the next sentence due tofailure of correct speech input to some extent despite last input ofcorrect speech, the recording data generated upon last input of thecorrect speech may be replayed to allow the learner to proceed to thenext sentence after speaking after the correct speech.

According to the present invention, when the content of the speech inputby the learner is not exactly the same as the learning target sentence,the speech input by the learner may be converted into text (hereinafter,referred to as inconsistent text) by the speech recognition unit in theterminal to be stored corresponding to the learning target sentence or asentence number thereof. Such inconsistent texts may be used asmaterials for foreign language learning of the learner.

According to the present invention, such inconsistent texts may beextracted to be provided as audio or video content to the learner toallow the learner to make an effort to input correct speech afterchecking a mistake in the content of the speech input by the learner.

According to the present invention, the inconsistent texts stored in thestorage device may be rated according to occurrence frequency and partor the entirety of the inconsistent text may be provided to the learnersuch that the learner can check ranks of the inconsistent texts and canmake an effort to avoid frequent mistakes.

In one embodiment of the present invention, words may be extracted froma sentence that the learner fails to input correct speech and may bechecked by the learner as to whether the words are included in textgenerated through speech recognition such that words not included in thetext generated through speech recognition can be provided to the learnerthrough a TTS sound or through the screen of the terminal. The followingdescription will focus on this procedure.

When the learner speaks a current learning target sentence to theterminal (in the learning mode described above) and when the learnerspeaks the next learning target sentence (in the conversation modedescribed above), each of the learning target sentences may be comparedwith text generated by converting the speech input by the learnerthrough the speech recognition function of the terminal. Then, upondetermining that the text is exactly same as the corresponding learningtarget sentence, words may be extracted from the learning targetsentence and may be checked as to whether the words are included in thetext generated through speech recognition such that words not includedin the text generated through speech recognition can be provided to thelearner through a TTS sound or through the screen of the terminal. Thewords not included in the text may be stored as incorrect wordscorresponding to the sentence number of the learning target sentence inan internal storage device of the terminal or in an external storagedevice. The terminal may extract frequency of the incorrect words, mayarrange the incorrect words in the order of words with high frequencythereof, and may notify all or part of the words to the learner.

(11) Detailed Description and Examples of a Solution to the EleventhProblem (Solution to Copyright Law)

According to the present invention, in order to prevent implementationof the present invention from becoming illegal, content installed withpermission of a copyright holder may be searched for in the terminal orin a specific folder inside the terminal in the course of executing appsoftware upon implementation of the present invention using contentsubject to copyright law. If such content is not found, the methodaccording to the present invention may further include a process ofsuspending execution of the content installed without permission of thecopyright holder and displaying a message requesting execution of thecontent after installing the content obtained with permission of thecopyright holder. Details of the processes of confirming whether foreignlanguage content installed under permission of a copyright holder ispresent in the terminal or in a specific folder inside the terminal,suspending execution of the content installed without permission of thecopyright holder, and displaying a message requesting execution of thecontent after installing the content obtained with permission of thecopyright holder are well known in the art and detailed descriptionthereof will be omitted.

(12) Detailed Description and Examples of a Solution to the EleventhProblem (Use of Multiple Subtitles)

For foreign language learning, since it is necessary for learners tounderstand the meaning of a foreign language to be learned, there is aneed for subtitles in a native language. Such subtitles are required tobe separately produced using a language used by each of the learners.Accordingly, a foreign language learning machine and a foreign languagelearning method for learners from various countries are preparedseparately according to the language used by each of the learners.However, when the learning machine or the learning method is aimed atlearning the same foreign language and is provided with a plurality ofsubtitles created in different languages, a plurality of learnerscapable of speaking the same language as the subtitles can use thesubtitles together and there is no need for separate production of thesubtitles with each language, thereby providing economic feasibility

Furthermore, the learning machine or the learning method aimed atlearning the same foreign language (first foreign language) may beprovided with a plurality of subtitles produced in different languagessuch that the subtitles produced in a foreign language (second foreignlanguage) other than a native language of the learner can be displayedon the screen of the terminal according to learner selection and can beread to the learner with a TTS sound by the terminal so as to allow thelearner to speak after the terminal, whereby the learner can effectivelylearn multiple foreign languages without using a separate foreignlanguage learning machine or method.

When the subtitles created in a plurality of languages are displayedtogether on the screen, the meaning of a foreign language to be learnedcan be understood more clearly by the learner and there is an effect ofmaking it possible for the learner to recognize errors in the incorrectsubtitles.

The method of producing and displaying the plurality of subtitles on thescreen of the terminal according to learner selection may be realized byproducing a plurality of subtitles in different foreign languagesthrough translation of learning target foreign language sentences suchthat the learner can select subtitles in one or multiple foreignlanguages together with native language subtitles on the screen of theterminal and the size of characters can be adjusted according to thenumber of subtitles selected by the learner.

In order to use the language of the subtitles for learning of a secondforeign language, the terminal is required to read the subtitlesselected by the learner using the TTS function. However, there is aproblem that the terminal fails to read a short sentence due tolimitation of the TTS function. In this case, the problem may be solvedby linking the content of subtitles twice or more to transform the shortsentence into a sentence having a sufficient length to read the sentencewith the TTS. If the length of the sentence is too short to read thesentence with the TTS even after linking the sentence twice, thesentence may be liked three times. Since the learner is guided to repeatspeaking of the content of the subtitles after listening to the contentof the subtitles read to the learner by the terminal, the number of suchconnections may be adjusted according to the length of the subtitles andreading ability of the TTS function without being limited to aparticular number of connections.

Depending on characteristics of a foreign language and learning abilityof the learner, foreign language learning cannot be efficientlyperformed since it is very difficult for the learner to speak the entirecontent of the subtitles after listening to the entire subtitle sentenceonce. In this case, more efficient learning can be achieved when theterminal reads only a certain number of words or characters in aspecific subtitle sentence to the learner with a TTS sound to allow thelearner to speak after the terminal. In particular, in cases where it isimportant and difficult to learn pronunciation of each letter in acertain language, such as Chinese, preferably, the terminal reads only apart of each sentence (as many words or characters as the numberselected by a user) to the learner with a TTS sound to allow the learnerto speak the corresponding part after the terminal, instead of readingthe entire sentence such that the learner speaks the entire sentenceafter listening to the entire sentence from the beginning. Since it isnot difficult in terms of character processing technology to extract andread a certain number of words or characters from a sentence, detaileddescription of this technology is omitted herein.

Although some embodiments have been described herein, it should beunderstood that these embodiments are provided for illustration only andare not to be construed in any way as limiting the present invention,and that various modifications, changes, alterations, and equivalentembodiments can be created by those skilled in the art without departingfrom the spirit and scope of the invention.

INDUSTRIAL APPLICABILITY

The present invention relates to a conversation-based foreign languagelearning method that enables foreign language learning through naturalconversation between a learner and a terminal, such as a smartphone, atablet computer, a notebook computer, a talking toy, an AI speaker, andthe like, based on video or audio content containing foreign languagesound expressions, such as movies, dramas, pop songs, news, and thelike, via speech transmission using a speech recognition function and aTTS function of the terminal, and may be very usefully applied toforeign language learning industries.

1. A conversation-based foreign language learning method using a speechrecognition function and a TTS function of a terminal throughinteractive conversation between a learner and the terminal based onvideo or audio foreign language content containing a plurality offoreign language learning target expressions, the method comprising: aprocess of repeating an operation that the terminal informs the learnerof a speech waiting state through a sound signal and/or a screen thereofwith an image indicating the speech waiting state of the terminal and,when the learner inputs speech to the terminal in the speech waitingstate, the terminal responds according to the speech input by thelearner and enters the speech waiting state again, followed by informingthe learner of the speech waiting state through the sound signal and/orthe screen to allow the learner to input new speech to the terminal,whereby the learner can perform foreign language learning throughinteractive conversation with the terminal via speech transmissionwithout touch of, if any, the screen or by minimizing touch of thescreen to exceptional cases; a process in which the terminal previouslystores voice commands to be delivered from the learner to the terminalin a storage device, such as a server or the cloud; when the learnerinputs speech in the speech waiting state of the terminal, the terminalexecutes a corresponding command, enters the speech waiting state againand informs the learner of the speech waiting state upon determiningthat the speech input by the learner matches a specific command storedin the storage device through comparison of the speech with the commandsstored in the storage device using a speech recognition function; theterminal determines whether the speech input by the learner is the sameas a current learning target sentence in the foreign language content orbelongs to the same category as the current learning target sentenceupon determining that the speech input by the learner does not match anycommand stored in the storage device; the terminal reads the nextlearning target sentence of the current learning target sentence in theforeign language content with a TTS sound or a corresponding contentsound of the foreign language content, enters the speech waiting stateagain and informs the learner of the speech waiting state upondetermining that the speech input by the learner is the same as thecurrent learning target sentence or belongs to the same category as thecurrent learning target sentence; and the terminal reads the currentlearning target sentence with a TTS sound or a corresponding contentsound of the foreign language content, enters the speech waiting stateagain and informs the learner of the speech waiting state to allow thelearner to speak the current learning target sentence again upondetermining that the speech input by the learner is not the same as thecurrent learning target sentence or does not belong to the same categoryas the current learning target sentence; and a process in which, upondetermining through comparison of the speech with the commands stored inthe storage device that the speech input by the learner does not matchany command stored in the storage device, the terminal determineswhether the speech input by the learner is the same as the next learningtarget sentence of the current learning target sentence or belongs tothe same category as the next learning target sentence; and the terminalproceeds to the next sentence of the corresponding sentence, that is, asentence after the next learning target sentence of the current learningtarget sentence, reads the next sentence to the learner with a TTS soundor a corresponding content sound of the foreign language content, entersthe speech waiting state again and informs the learner of the speechwaiting state to allow the learner to input speech with respect to thenext sentence of the corresponding sentence, that is, a sentence afterthe next learning target sentence of the current learning targetsentence, upon determining that the speech input by the learner is thesame as the next learning target sentence or belongs to the samecategory as the next learning target sentence.
 2. The conversation-basedforeign language learning method according to claim 1, wherein a periodof time until the terminal enters the speech waiting state is set inconsideration of a length of a corresponding foreign language learningtarget sentence and/or a period of time for which the correspondingforeign language learning target sentence is executed in the foreignlanguage content.
 3. The conversation-based foreign language learningmethod according to claim 1, wherein, in determination as to whethertext obtained through conversion of the speech input by the learner isthe same as text of the foreign language learning target sentence in theforeign language content or belongs to the same category as the text ofthe foreign language learning target sentence, both texts are comparedassuming that punctuation marks, such as a question mark (?), anexclamation point (!), a comma (,), a period (.), a quotation mark (“”), a hyphen (-), a center point (⋅), parentheses {( )}, curly braces ({}), square brackets ([ ]), or spaces incorrectly added at the beginningand end of sentences, and spaces overlapping each other are absent; andthat an expression of numbers in Arabic numerals and an expression ofnumbers with letters; an expression of following verbs or auxiliaryverbs in abbreviated forms using an apostrophe (') after a subject andan expression of original forms thereof without using abbreviations; anexpression of negative sentences in abbreviated forms using anapostrophe (') and an expression of original forms thereof without usingabbreviations; “going to” and “gonna”; “want to” and “wanna”; “got to”and “gotta”; or a sentence used by distinguishing between uppercase andlowercase characters and a sentence used without distinctiontherebetween are regarded as being exactly or partially identical toeach other.
 4. The conversation-based foreign language learning methodaccording to claim 1, wherein, when learning with respect to a foreignlanguage learning target sentence is repeated due to failure of thelearner to continuously input speech that belongs to the same categoryas the foreign language learning target sentence or when the learnerinputs simple speech, such as “yes” or “okay”, or a preset voicecommand, such as “you speak the next learning target sentence”, theterminal stops reading of a current foreign language learning targetsentence and proceeds to the next learning target sentence of thecurrent foreign language learning target sentence to allow the learnerto learn the next learning target sentence of the current foreignlanguage learning target sentence.
 5. The conversation-based foreignlanguage learning method according to claim 1, wherein a terminal havinga screen displays a current learning target sentence of the foreignlanguage content together with the next learning target sentencethereof.
 6. The conversation-based foreign language learning methodaccording to claim 1, further comprising: a process in which theterminal reads a predetermined number of words or characters of aspecific sentence to the learner with a TTS sound among learning targetsentences, enters the speech waiting state and informs the learner ofthe speech waiting state to allow the learner to speak the predeterminednumber of words or characters of the specific sentence after theterminal; and the terminal proceeds to the next learning targetsentence, reads a predetermined number of words or characters of thenext learning target sentence to the learner with a TTS sound, entersthe speech waiting state and informs the learner of the speech waitingstate to allow the learner to speak the predetermined number of words orcharacters of the next learning target sentence after the terminal, upondetermining that speech input by the learner is the same as thepredetermined number of words or characters of the specific sentence orbelongs to the same category as the predetermined number of words orcharacters of the specific sentence, and the terminal reads the samepart of the specific sentence again to the learner with the TTS sound toallow the learner to speak the same part of the specific sentence againafter the terminal, upon determining that the speech input by thelearner is not the same as the predetermined number of words orcharacters of the specific sentence or does not belong to the samecategory as the predetermined number of words or characters of thespecific sentence.
 7. The conversation-based foreign language learningmethod according to claim 1, wherein the terminal reads native languagesubtitles to the learner with a TTS sound corresponding to the learningtarget sentence before the terminal provides a foreign language contentsound or a TTS sound for the learning target sentence to the learner. 8.The conversation-based foreign language learning method according toclaim 1, wherein the terminal reads the learning target sentence to thelearner with a TTS sound before the terminal provides a foreign languagecontent sound for the learning target sentence to the learner.
 9. Theconversation-based foreign language learning method according to claim1, wherein the terminal reads the learning target sentence and nativelanguage subtitles to the learner with TTS sounds corresponding to thelearning target sentence before the terminal provides a foreign languagecontent sound for the learning target sentence to the learner.
 10. Theconversation-based foreign language learning method according to claim1, wherein, when the speech waiting state of the terminal is lost, thelearner is guided to press one or more buttons on a remote controller orearphones, or to use another software program for speech recognition, soas to activate the speech waiting state.
 11. The conversation-basedforeign language learning method according to claim 1, whereinperformance of foreign language learning of the learner is evaluated andthe learner is informed thereof in a process of allowing the learner tolearn a foreign language through interactive conversation using theforeign language content.
 12. The conversation-based foreign languagelearning method according to claim 1, wherein, when the terminalproceeds to the next learning target sentence to perform foreignlanguage learning based on determination that the speech input by thelearner is exactly the same as the current learning target sentence,speech recording data of the learner are stored corresponding to thelearning target sentence such that the speech recording data generatedby exactly speaking a specific sentence can be preserved and used asneeded.
 13. The conversation-based foreign language learning methodaccording to claim 12, comprising: at least one selected from the groupconsisting of a process of recording and storing data obtained byexactly speaking a learning target sentence by the learner to use thedata as data for improving speed and accuracy of speech recognition incomparison with a content sound for the corresponding sentence, aprocess of recording and storing data obtained by exactly speaking alearning target sentence by the learner to use the data as data forimproving speed and accuracy of speech recognition in comparison withprevious recording data of the learner, a process of recording andstoring data obtained by exactly speaking a learning target sentence bythe learner to use the data as assistant data in comparison with datafor speech recognition stored in the storage device by a speechrecognition system, and combinations thereof.
 14. The conversation-basedforeign language learning method according to claim 1, wherein, upondetermining that speech input by the learner is not exactly the same asa learning target sentence, text converted from the speech of thelearner by a speech recognition function of the terminal is storedcorresponding to the learning target sentence or a sentence numberthereof.
 15. The conversation-based foreign language learning methodaccording to claim 1, wherein, upon determining that speech input by thelearner is not exactly the same as a learning target sentence, thelearner is informed of a difference between text converted from thespeech of the learner by a speech recognition function of the terminaland the learning target sentence.
 16. The conversation-based foreignlanguage learning method according to claim 1, wherein, upon determiningthat speech input by the learner is not exactly the same as a learningtarget sentence, a difference between text converted from the speech ofthe learner by a speech recognition function of the terminal and thelearning target sentence is stored in the terminal.
 17. Theconversation-based foreign language learning method according to claim1, wherein, upon determining that speech input by the learner is notexactly the same as a learning target sentence, the learner is informedof words not included in text converted from the speech of the learnerby a speech recognition function of the terminal among words included inthe learning target sentence.
 18. The conversation-based foreignlanguage learning method according to claim 1, wherein, upon determiningthat speech input by the learner is not exactly the same as a learningtarget sentence, words not included in text converted from the speech ofthe learner by a speech recognition function of the terminal among wordsincluded in the learning target sentence are stored in the terminal. 19.The conversation-based foreign language learning method according toclaim 1, wherein, upon determining that speech input by the learner isnot exactly the same as a learning target sentence, words not includedin text converted from the speech of the learner by a speech recognitionfunction of the terminal among words included in the learning targetsentence are stored in the terminal, a frequency of each of the wordsstored in the terminal is checked, and the learner is informed of all orsome of words with high frequency among the words stored in theterminal.
 20. The conversation-based foreign language learning methodaccording to claim 1, wherein an identifier is added to all or part ofthe voice commands delivered from the learner to the terminal so as toprevent confusion with the learning target sentence.
 21. Theconversation-based foreign language learning method according to claim1, further comprising: a process of displaying a message asking toexecute foreign language learning after storing foreign language contentwith permission of a copyright holder in the storage device, followed byprohibiting foreign language learning upon determining that there is noforeign language content with permission of the copyright holder in thestorage device.
 22. A conversation-based foreign language learningmethod using a speech recognition function and a TTS function of aterminal through interactive conversation between a learner and theterminal based on video or audio foreign language content containing aplurality of foreign language learning target expressions, the methodcomprising: a process of repeating an operation that the terminalinforms the learner of a speech waiting state through a sound signaland/or a screen thereof with an image indicating the speech waitingstate of the terminal and, when the learner inputs speech to theterminal in the speech waiting state, the terminal responds according tothe speech input by the learner and enters the speech waiting stateagain, followed by informing the learner of the speech waiting statethrough the sound signal and/or the screen to allow the learner to inputnew speech to the terminal, whereby the learner can perform foreignlanguage learning through interactive conversation with the terminal viaspeech transmission without touch of, if any, the screen or byminimizing touch of the screen to exceptional cases; a process in whichthe terminal previously stores voice commands to be delivered from thelearner to the terminal in a storage device, such as a server or thecloud; when the learner inputs speech in the speech waiting state of theterminal, the terminal executes a corresponding command, enters thespeech waiting state again and informs the learner of the speech waitingstate upon determining that the speech input by the learner matches aspecific command stored in the storage device through comparison of thespeech with the commands stored in the storage device using a speechrecognition function; the terminal determines whether the speech inputby the learner is the same as a current learning target sentence in theforeign language content or belongs to the same category as the currentlearning target sentence upon determining that the speech input by thelearner does not match any command stored in the storage device; theterminal reads the next learning target sentence of the current learningtarget sentence in the foreign language content with a TTS sound or acorresponding content sound of the foreign language content, enters thespeech waiting state again and informs the learner of the speech waitingstate upon determining that the speech input by the learner is the sameas the current learning target sentence or belongs to the same categoryas the current learning target sentence; and the terminal reads thecurrent learning target sentence with a TTS sound or a correspondingcontent sound of the foreign language content, enters the speech waitingstate again and informs the learner of the speech waiting state to allowthe learner to speak the current learning target sentence again upondetermining that the speech input by the learner is not the same as thecurrent learning target sentence or does not belong to the same categoryas the current learning target sentence; a process in which, upondetermining through comparison of the speech with the commands stored inthe storage device that the speech input by the learner does not matchany command stored in the storage device, the terminal determineswhether the speech input by the learner is the same as the next learningtarget sentence of the current learning target sentence or belongs tothe same category as the next learning target sentence; the terminalproceeds to the next sentence of the corresponding sentence, that is, asentence after the next learning target sentence of the current learningtarget sentence, reads the next sentence to the learner with a TTS soundor a corresponding content sound of the foreign language content, entersthe speech waiting state again and informs the learner of the speechwaiting state to allow the learner to input speech with respect to thenext sentence of the corresponding sentence, that is, a sentence afterthe next learning target sentence of the current learning targetsentence, upon determining that the speech input by the learner is thesame as the next learning target sentence or belongs to the samecategory as the next learning target sentence; and a process in which,upon determining that the speech input by the learner is the same as anyone sentence excluding the current learning target sentence and the nextlearning target sentence among learning target sentences, the terminalproceeds to the next sentence of the corresponding sentence, reads thenext sentence to the learner with a TTS sound or a corresponding contentsound of the foreign language content, enters the speech waiting stateagain and informs the learner of the speech waiting state to allow thelearner to input speech with respect to the sentence read to the learnerby the terminal or the next sentence thereof, that is, the next sentenceof the sentence read to the learner by the terminal.
 23. Aconversation-based foreign language learning method using a speechrecognition function and a TTS function of a terminal throughinteractive conversation between a learner and the terminal based onvideo or audio foreign language content containing a plurality offoreign language learning target expressions, the method comprising: aprocess of repeating an operation that the terminal informs the learnerof a speech waiting state through a sound signal and/or a screen thereofwith an image indicating the speech waiting state of the terminal and,when the learner inputs speech to the terminal in the speech waitingstate, the terminal responds according to the speech input by thelearner and enters the speech waiting state again, followed by informingthe learner of the speech waiting state through the sound signal and/orthe screen to allow the learner to input new speech to the terminal,whereby the learner can perform foreign language learning throughinteractive conversation with the terminal via speech transmissionwithout touch of, if any, the screen or by minimizing touch of thescreen to exceptional cases; a process in which the terminal previouslystores voice commands to be delivered from the learner to the terminalin a storage device, such as a server or the cloud; and when the learnerinputs speech in the speech waiting state of the terminal, the terminalexecutes a corresponding command upon determining that the speech inputby the learner matches a specific command stored in the storage devicethrough comparison of the speech with the commands stored in the storagedevice using a speech recognition function; a process in which, upondetermining that the speech input by the learner does not match anycommand stored in the storage device, the terminal determines whetherthe speech input by the learner is the same as a current learning targetsentence in the foreign language content or belongs to the same categoryas the current learning target sentence; the terminal reads the nextlearning target sentence of the current learning target sentence in theforeign language content with a TTS sound or a corresponding contentsound of the foreign language content and enters the speech waitingstate again upon determining that the speech input by the learner is thesame as the current learning target sentence or belongs to the samecategory as the current learning target sentence; and the terminal readsthe current learning target sentence with a TTS sound or a correspondingcontent sound of the foreign language content, enters the speech waitingstate again and informs the learner of the speech waiting state to allowthe learner to speak the current learning target sentence again upondetermining that the speech input by the learner is not the same as thecurrent learning target sentence or does not belong to the same categoryas the current learning target sentence; a process in which, upondetermining through comparison of the speech with the commands stored inthe storage device that the speech input by the learner does not matchany command stored in the storage device, the terminal determineswhether the speech input by the learner is the same as the next learningtarget sentence of the current learning target sentence or belongs tothe same category as the next learning target sentence; the terminalproceeds to the next sentence of the corresponding sentence, that is, asentence after the next learning target sentence of the current learningtarget sentence, reads the next sentence to the learner with a TTS soundor a corresponding content sound of the foreign language content, entersthe speech waiting state again and informs the learner of the speechwaiting state to allow the learner to input speech with respect to thenext sentence of the corresponding sentence, that is, a sentence afterthe next learning target sentence of the current learning targetsentence, upon determining that the speech input by the learner is thesame as the next learning target sentence or belongs to the samecategory as the next learning target sentence; and a process in whichthe terminal is provided with a plurality of subtitles created indifferent languages including a native language and corresponding tolearning target foreign language sentences such that one or pluralsubtitles can be displayed on a screen of the terminal according tolearner selection.
 24. The conversation-based foreign language learningmethod according to claim 23, further comprising: a process in which theterminal reads the subtitles to the learner with a TTS sound in aforeign language other than the native language of the learner accordingto learner selection.
 25. The conversation-based foreign languagelearning method according to claim 24, further comprising: a process inwhich, after reading the subtitles to the learner using the TTS sound inthe foreign language other than the native language of the learner, theterminal enters the speech waiting state and informs the learner of thespeech waiting state to allow the learner to speak what the learnerlistens to; the terminal reads the next subtitles of the currentsubtitles to the learner with a TTS sound and enters the speech waitingstate upon determining that speech input by the learner is the same asthe content of the subtitles read by the terminal or belongs to the samecategory as the content of the subtitles read by the terminal; and theterminal reads the current subtitles with a TTS sound, enters the speechwaiting state and informs the learner of the speech waiting state toallow the learner to speak the current subtitles again after theterminal, upon determining that the speech input by the learner is notthe same as the content of the subtitles read by the terminal or doesnot belong to the same category as the content of the subtitles read bythe terminal.
 26. The conversation-based foreign language learningmethod according to claim 24, wherein, when the terminal reads thesubtitles to the learner using the TTS sound in the foreign languageother than the native language of the learner and a correspondingsentence of the subtitles is too short for the terminal to read thecorresponding sentence, the corresponding sentence is linked twice ormore to a sentence having a length so as to allow the terminal to readthe corresponding sentence.
 27. The conversation-based foreign languagelearning method according to claim 24, wherein the terminal reads a partof a specific sentence to the learner with a TTS sound instead ofreading the entirety of the specific sentence among sentences of thesubtitles in foreign languages other than the native language.
 28. Theconversation-based foreign language learning method according to claim1, wherein a process of sequentially displaying learning targetsentences of the foreign language content on the screen of the terminalis implemented using a text switcher function supported by AndroidStudio and Java; native language subtitles corresponding to the learningtarget sentences of the foreign language content are stored in the formof HashMap of Java and an Android Studio program to be used as needed;and a process of executing an audio file containing a content sound ofthe foreign language content is implemented using a sound pool functionof the Android Studio program and Java.
 29. (canceled)